ASM reported that a FRA diskgroup was 100% full, but when running du's from asmcmd or selecting the file usage from v$asm_file, it was only showing a few hundred MB in use:
SYS +ASM> select sum(total_mb) total_mb, sum(free_mb) free_mb from v$asm_disk where group_number in (select group_number from v$asm_diskgroup where name='RECO'); TOTAL_MB FREE_MB ---------- ---------- 51200 0 SYS +ASM> SELECT gname, dbname, file_type, round(SUM(space)/1024/1024) mb, round(SUM(space)/1024/1024/1024) gb, COUNT(*) "#FILES" FROM ( SELECT gname, regexp_substr(full_alias_path, '[[:alnum:]_]*',1,4) dbname, file_type, space, aname, system_created, alias_directory FROM ( SELECT concat('+'||gname, sys_connect_by_path(aname, '/')) full_alias_path, system_created, alias_directory, file_type, space, level, gname, aname FROM ( SELECT b.name gname, a.parent_index pindex, a.name aname, a.reference_index rindex , a.system_created, a.alias_directory, c.type file_type, c.space FROM v$asm_alias a, v$asm_diskgroup b, v$asm_file c WHERE a.group_number = b.group_number AND a.group_number = c.group_number(+) AND a.file_number = c.file_number(+) AND a.file_incarnation = c.incarnation(+) ) START WITH (mod(pindex, power(2, 24))) = 0 AND rindex IN ( SELECT a.reference_index FROM v$asm_alias a, v$asm_diskgroup b WHERE a.group_number = b.group_number AND ( mod(a.parent_index, power(2, 24))) = 0 ) CONNECT BY prior rindex = pindex ) WHERE NOT file_type IS NULL and system_created = 'Y' ) WHERE gname='RECO' GROUP BY gname, dbname, file_type ORDER BY gname, dbname, file_type / GNAME DBNAME FILE_TYPE MB GB #FILES ---------------- ---------- -------------- ---------- ---------- ---------- RECO DB1 ARCHIVELOG 6 0 1 RECO DB1 CONTROLFILE 24 0 1 RECO DB1 ONLINELOG 408 0 8 RECO DB2 ARCHIVELOG 1 0 1 RECO DB2 CONTROLFILE 16 0 1 RECO DB2 ONLINELOG 153 0 3
(script is a modified version from http://ronr.blogspot.com/2013/07/listing-asm-storage-usage-per-diskgroup.html)
Something had gotten corrupt in the diskgroup, and the simple fix was to run:
SYS +ASM> alter diskgroup oem_dg_flash check all repair; Diskgroup altered. Elapsed: 00:00:00.10 SYS +ASM> select sum(total_mb) total_mb, sum(free_mb) free_mb from v$asm_disk where group_number in (select group_number from v$asm_diskgroup where name='RECO'); TOTAL_MB FREE_MB ---------- ---------- 51200 50535
